This example project describes how to create a serial data logger using a Raspberry Pi. The example provides a Python script that is automatically started at power-on to receive serial data from an external source using the 'serial0' UART input, and then timestamps and logs the data to an HDMI display and a connected USB drive. This example uses an RPi mounted to the back of a SunFounder 10" LCD display, however any HDMI display can be used. This example uses an MSP430 development board to stream serial data, however alternate serial data sources can also be used.
This example used a Raspberry Pi 3 Model B mounted to the back of a SunFounder 10" LCD display, as shown below. The example setup also included a 5V 2.5A power supply, and a standard wireless keyboard and mouse combo. In order to log data a USB drive is also needed. I used a SanDisk Cruzer Fit 8GB drive. Any USB drive should work, but the small form factor of this drive hides nicely behind the display and provides a clean looking setup.
SunFounder 10" LCD Display with Raspberry Pi 3 Model B |
+
|
SanDisk Cruzer Fit USB Drive or similar |
IMPORTANT: Make sure the Raspberry Pi has been made "Project Ready" before proceeding.
IMPORTANT: Make sure the USB Drive has been formatted as FAT (e.g. FAT32), and that the drive is given the label "USBDRIVE", and that a textfile named "validate.txt" is created at the top level of the drive. The reason why this is necessary will be explained later.
The example project uses serial data coming into the UART RX (Serial Input) pin 10. When the Raspberry Pi is mounted on the back of the LCD display, this is the 5th pin from the left on the top row. It is marked by a yellow dot on the image below. This pin must be connected to the serial output of the device that will provide the data to be logged. A ground reference is also required. That connection was made using the pin 6 ground. This is the 3rd pin from the left on the top row, and is marked with a green dot in the image below.
In order to provide a stream of serial data for the example, the MSP430 4-bit counting example was modified to include the MSP430F5529LP_UART library, and to assemble and transmit a serial message in the main loop each time the counter was incremented. The code added to the example looked like this:
sprintf(string, "%3d, %3X, %d%d%d%d%d%d%d%d \r\n", s_count_u8, s_count_u8, BYTETOBINARY(s_count_u8)); SendSerialMsg(string, strlen(string)); |
The format of the data is not important, although comma-separated data makes it easy to import into applications such as Excel. The important part is that the string is terminated by a '\n' new line character. This informs the Python script running on the Raspberry Pi that the data stream is complete which triggers the logging of the bytes received. The '\r' carriage return character is ignored by the script and is not captured or logged. It is included to make the data stream compatible with both Windows and Linux.
The datalogger.py Python script looks like this:
#!/usr/bin/env python """ /* ######################################################################### */ /* * This file was created by www.DavesMotleyProjects.com * * This software is provided under the following conditions: * Permission is hereby granted, free of charge, to any person obtaining * a copy of this software and associated documentation files (the * 'Software'), to deal in the Software without restriction, including * without limitation the rights to use, copy, modify, merge, publish, * distribute, sublicense, and/or sell copies of the Software, and to * permit persons to whom the Software is furnished to do so, subject to * the following conditions: * * THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. * IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY * CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, * TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. * */ /* ######################################################################### */ """ import io import os import time import datetime import serial from time import sleep # to create delays from locale import format # to format numbers from shutil import rmtree # to remove directories PathValid = False FileName = "" FilePath = "" dataIndex = 0 FAIL_PATH = "/media/pi/USBDRIVE1/" DRIVE_PATH = "/media/pi/USBDRIVE/" AUTH_PATH = DRIVE_PATH + "validate.txt" MAX_ROWS_IN_FILE = 65535 """############################################################################ Function: chk_usb_on_start Description: This function is called at the start of the program execution and provides a helpful message to the user on the approprate actions to take to start datalogging, if a valid usb drive was not found. ############################################################################""" def chk_usb_on_start(): global PathValid, DRIVE_PATH, AUTH_PATH # if the "validate.txt" file is not found on startup, prompt the user to # install the appropriate usb drive if not (os.path.exists(AUTH_PATH)): print "\n" print "To start datalogging, insert a usb drive that is formatted as FAT, with the " print "label 'USBDRIVE', that has a text file named 'validate.txt' at the top level " print "of the usb drive.\n" while not (os.path.exists(AUTH_PATH)): sleep(1) if (os.path.isdir(DRIVE_PATH)): if (os.path.isdir(FAIL_PATH)): PathValid=True validate_usb_write() PathValid=False print "To finish path correction, remove and replace USBDRIVE." print "'USBDRIVE' with 'validate.txt' file found." """############################################################################ Function: set_path_invalid Description: This function is called whenever a path verification check has failed in order to inform the user of the error, and to set the PathValid variable to false to prevent any further attempts to write to the usb drive until the error condition is corrected. ############################################################################""" def set_path_invalid(): global PathValid, dataIndex # if the path is already defined as invalid, do nothing, else, set the # path as invalid, inform the user, and reset the dataIndex. if not PathValid == False: PathValid = False print "USB path is not valid, corrupted, missing, or ejected" dataIndex = 0 """############################################################################ Function: validate_usb_write Description: This function checks for the creation of an erroneous USB drive path, which can occur rarely if the USB drive is removed, without ejecting it first, and the removal occurs between the code that checks for a valid path, and where the data is written to the USB drive. If the USB doesn't exist, and the write occurs, a 'fake" USBDRIVE owned by root will be created and data logging will continue to that location, and all future insertions of the USB will be assigned USBDRIVE1, and data will not be written to the USB drive again. When this condition is detected, the condition will be automatically corrected by erasing the fake path. IMPORTANT: to detect this condition a text file called "validate.txt" needs to be placed on the USB drive. ############################################################################""" def validate_usb_write(): global PathValid, DRIVE_PATH, AUTH_PATH # if we already know the path isn't valid, this check isn't needed. if PathValid == False: return # if the "validate.txt" file is not found, then we may have created a # 'fake' USB drive path accidentally... if not (os.path.exists(AUTH_PATH)): print "path corruption suspected" sleep(1) # The sleep above is to provide some manner of debouncing, in case # the detection error of "validate.txt" was a timing / race condition. # After all, the result could be erasing all of our data, so it pays # to double-check. # if the "validate.txt" file is not found a second time... if not (os.path.exists(AUTH_PATH)): print "path corruption confirmed" print "validate.txt file was not found" set_path_invalid() # remove the drive path. the location that is being written to is # not the USB drive. This happens rarely, when attempting a write # to a USB that doesn't exist. Linux will create a temporary # location, and will appear to be logging to the USB, but isn't. rmtree(DRIVE_PATH, ignore_errors = True) # if the path no longer exists then rmtree worked, and the # incorrect path was deleted. if not (os.path.isdir(DRIVE_PATH)): print "path corruption corrected" """############################################################################ Function: start_new_file Description: This function is called when a new file is needed to save data logging results. A filename is assembled based on the current Date and Time, and if the creation of the file is successful, the file is initialized with a header row. If all was successful, PathValid will be set to True (This is the only location where this is set to True) ############################################################################""" def start_new_file(): global PathValid, FileName, FilePath, dataIndex # Assemble the filename based on the current date and time. Then assign # the FilePath to start using the new file. FileName = str("_".join(str(datetime.datetime.now()).split(" "))) FileName = str("_".join(FileName.split(":"))) FileName = str("_".join(FileName.split("."))) FileName = str("_".join(FileName.split("-"))) FileName = FileName + ".csv" FilePath = DRIVE_PATH + FileName # if the drive path exists... if (os.path.isdir(DRIVE_PATH)): # create a new blank file. If this file existed, which it shouldn't, # open with 'w' would overwrite it. mFile = open(FilePath, "w") # display to the user that a new file has been started print print "#############################################################" print "New File: " + FileName print "#############################################################" print # write to the new file a header row to indicate what each of the # values represent. In this example, the 'Data' values are unknown. It # is expected that 'Data' is a string of an unknown number of values, # to which the proper csv formatting has been applied. If you know your # data formatting, and want to make this program specific, add your # data headers here. mFile.write("Index, Date, Time, Data \r\n") mFile.close() PathValid = True dataIndex = 1 # if the drive path didn't exist... else: set_path_invalid(); """############################################################################ Function: read_data Description: This function reads the serial input, assembling chars until the new line character is received. On reception of a new line, the function generates a timestamp, which is pre-pended to the received data, and the entire string is written to the display and logged to the usb drive (if a valid drive exists). ############################################################################""" def read_data(): global PathValid, FileName, FilePath, dataIndex, ser # if there is currently no file to write to, attempt to create a file. If # that fails, simply return. This means that there is no valid USB inserted # so there is no reason to attempt writing to a file. if (PathValid == False): start_new_file() if (PathValid == False): return valueString = "" # reset the string that will collect chars mchar = ser.read() # read the next char from the serial port # continue reading characters and appending them to the valueString # (ignoring carriage returns) until a 'new line' character is received. while (mchar != '\n'): if (mchar != '\r'): valueString += mchar mchar = ser.read() # after a full valueString has been assembled, create the timestamp millis = int(round(time.time() * 1000)) rightNow = str(datetime.datetime.now()).split() mDate = rightNow[0] mTime = rightNow[1] # format the full string: index, timestamp, and data fileString = str(format('%05d', dataIndex)) + ", " + \ str(mDate) + ", " + str(mTime) + ", " + valueString # if execution reaches this point, then it is assumed that a valid USB is # inserted, and a file exists to write the data. Open the data file with # 'a' to append the new data, write the data string, close the file, and # increment the index. try: # before writing, double-check that the data logging file exists if (os.path.exists(FilePath)): print(fileString) fileString += "\r\n" mFile = open(FilePath, "a", 1) mFile.write(fileString) mFile.close() dataIndex += 1 validate_usb_write() # if the file doesn't exist, but the drive path still exsists, then it # is possible that the file was deleted while in use. elif (os.path.isdir(DRIVE_PATH)): start_new_file() else: set_path_invalid() except: print("write failed") # if the number of rows written to the data file exceeds MAX_ROWS_IN_FILE # start a new file. This was added to address an issue where Excel would # not import more than 65535 rows from a csv file, when the csv is opened # directly, even though Excel is supposed to have a maximum number of rows # closer to 1048576. if (dataIndex >= MAX_ROWS_IN_FILE): start_new_file() """############################################################################ Function: main ############################################################################""" def main(): global ser ser = serial.Serial() ser.baudrate = 57600 ser.timeout = None ser.port = '/dev/serial0' print ser print ser.name ser.open() print "Serial port is open: ", ser.isOpen() chk_usb_on_start() start_new_file() try: while (True): read_data() finally: ser.close print "Serial port is open: ", ser.isOpen() return 0 """############################################################################ This next section checks to see whether the file is being executed directly. If it is, then __name__ == '__main__' will evaluate as True, and the main() function will execute. If the file is being imported by another module, then __name__ will be set to the modules name instead. ############################################################################""" if __name__ == '__main__': main() """############################################################################ End of File: datalogger.py ############################################################################""" |
Before setting up the automation, try installing and running the script manually. For this initial test, it is recommended that the USB drive be removed, and the serial data stream be disconnected before starting. If everything works, this should provide a nice 'static' display to observe, and if needed troubleshoot. To download and test the datalogger, open a terminal window, and perform the following actions.
Create a folder called datalogger in the pi home directory, and then enter the new directory.
Download the datalogger.py Python script file, by entering the following in the terminal window. The wget command will download the specified file, and place it in your current directory, which is the datalogger directory.
Now launch the script manually. Note: sudo is required in order to allow Python to access the Serial UART, which is a restricted resource.
If everything worked, you should see the following display. The serial port being used is /dev/serial0, which is a high-performance hardware UART, and when queried the port should respond as 'Serial port is open: True'. If the USB drive was not installed when the datalogger started, a user prompt to install a correctly configured USB drive will also be displayed.
If the Serial port isn't opening, verify that the RPi was made "Project Ready". If the Serial port is working but there appears to be something wrong with the script, you may want to consider opening and running the script inside of an editor like geany. You can do that by entering the following. Note: sudo is still required to access the serial port. The '&' at the end simply releases terminal window after executing the command. If you find an issue with the script, please provide feedback, and I will correct it.
If everything worked above, insert a properly configured USB drive (but don't connect the serial data source yet). The script should detect the existence of the drive, and will indicate that it was found, and a new file will be created and the file name will be displayed, as shown below.
Now connect the serial data source. Data logging should start, as shown in the window below.
When you open the corresponding .csv file on the USB drive, you will see the following data. (Only the first several lines are shown). The first row is added to the file each time a new file is created. This action is performed in the function start_new_file(). Currently everything beyond "Index, Date, Time" is listed as just "Data" because the script is currently generic. It doesn't know, or care, what data is passed to it or what format the data is in. If the script is customized for a specific purpose, you may want to consider changing the first row to include the actual data headers.
Index, Date, Time, Data 00001, 2017-02-19, 12:01:11.805133, 158, 9E, 10011110 00002, 2017-02-19, 12:01:12.325190, 159, 9F, 10011111 00003, 2017-02-19, 12:01:12.845139, 160, A0, 10100000 00004, 2017-02-19, 12:01:13.365236, 161, A1, 10100001 00005, 2017-02-19, 12:01:13.885333, 162, A2, 10100010 00006, 2017-02-19, 12:01:14.405431, 163, A3, 10100011 00007, 2017-02-19, 12:01:14.925677, 164, A4, 10100100 |
Once everything is working manually, modify the startup.sh script to automatically launch the data logging script at power-on.
In the terminal window, enter the following:
This will return to the /home/pi directory, and open the startup.sh script. In that file, enter the following items, save the file, and exit.
If they aren't already, insert the USB drive and connect the serial data source, then reboot the RPi by entering the following in the terminal window.
If everything works properly, the data logging will start automatically when the RPi restarts.
One of the things that may not be obvious, is why the requirement to label the USB drive as 'USBDRIVE', and why must it contain a file called 'validate.txt'?
The answer to the first question is that this allows us to know how the USB drive will be mounted. The file path will be '/media/pi/USBDRIVE'.
Answering the second question is a little more complicated. This has to do with the operating system response to executing the Python open(filepath, "a") command when the USB filepath doesn't exist. This can happen when the operating system un-mounts the drive just before the execution of the open command. In response, Raspbian will create a 'fake' USBDRIVE directory under /media/pi. When this occurs the Python script would normally continue data logging because the directory appears to exist, however the 'fake' USBDRIVE directory is not accessible. When the real USB drive is re-mounted, Raspbian will mount it as USBDRIVE1, and the data logging will be permanently broken.
To resolve this, the 'validate.txt' file requirement was added. The reason is that the 'validate.txt' file will not exist in the 'fake' USBDRIVE directory, and the condition can be detected and resolved by deleting the 'fake' path. This allows the real USB drive to be re-mounted correctly as USBDRIVE and data logging continues properly when re-inserted. The screenshot below shows the response when the 'fake' path is created by continuing to remove and insert the USB drive.
In order to keep the file size manageable, a new file is created each time the number of rows in the file reaches 65535. The file size is controlled by the value of MAX_ROWS_IN_FILE. An interesting question is, does the write speed slow down when appending to files that are very large? In order to test this, a small Python script was used to write 100000 lines to a text file on the USB drive, tracking the time before and immediately after creating the time stamp and data string, printing to the display, and opening, appending, and closing the USB text file. A plot of the resulting write speed is shown below. Although, it can be seen that occasionally the operating system will execute other tasks, interrupting the Python script, and extending the time, there is no apparent trend to suggest that write speed is dependent on file size.
The average write speed is about 4.5 ms.